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DETAILED ACTION 
Response to Amendment 

1 . In response to the office action from 12/1 1/2006, the applicants have submitted an 
amendment, filed 3/28/2007, amending claim 3 to overcome minor informalities, while arguing 
to traverse the art rejection based on the limitation regarding the claimed color/shape threshold 
storing means (Amendment, Pages 6-8). Applicants' arguments have been fully considered, 
however the previous rejection is maintained due to the reasons listed below in the response to 
arguments. 

2. In response to the amendment of Claim 3, the examiner has withdrawn the previous 
objection directed towards minor informalities. 



Response to Arguments 

3. Applicants' arguments have been fully considered but they are not persuasive for the 

i 

following reasons: 

With respect to the independent claims, the applicants argue that Hermes et al ("Image 
Retrieval for Information Systems, " 1995) fails to teach a "color/shape threshold retrieving 
means for retrieving the color histograms and the edge information corresponding to the 
analyzed words from the color/shape threshold storing means and retrieving an image satisfying 
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the retrieved color histograms and edge information." The applicants explain that their invention 
instead discloses that a query is compared to a histogram and edge information and then the 
histogram and edge information is used in retrieving an image (Amendment, Pages 6-7). 

In response the examiner notes that Hermes does teach a color/shape threshold storing 
means as set forth in the claimed invention. More specifically, Hermes discloses that color 
histograms and edge information associated with a particular image are mapped to particular 
words (Sections 2.1- 2 A 3, Section 3, and Figs. 6 and 8). When a search takes place, the mapped 
words are used to retrieve the corresponding histogram and edge information as a part of a 
desired image, thereby retrieving the image (Section 3 and Fig. 8). Thus, since Hermes discloses 
that histogram and contour information corresponding to a particular image are mapped to search 
terms for image retrieval, Hermes discloses the aforementioned claim limitation. 

From the applicants' arguments, it appears that it is intended to claim that a NL sentence 
is used to retrieve a color histogram and edge information, and it is this retrieved histogram and 
edge information that is compared or matched against images in a database to retrieve a 
particular image. It is noted, however, that no additional comparison step/means is clearly 
defined in the claims. The claims only define that an image is retrieved that satisfies such 
information. This claimed limitation is taught by Hermes, since, as is pointed out above, edge 
and histogram information associated with a particular image are mapped to search words. In 
response to applicant's argument that the references fail to show certain features of applicant's 
invention, it is noted that the features upon which applicant relies (i.e., a second matching step 
between color histogram and edge information and images in an image database) are not recited 
in the rejected claim(s). Although the claims are interpreted in light of the specification, 
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limitations from the specification are not read into the claims. See In re Van Geuns, 988 
F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). 

With respect to the Liu et al reference (U.S. Patent: 6,970,860), the applicants argue that 
Liu does not teach using histograms for text queries as the claimed invention does (Amendment, 
Page 7). In response, the examiner notes that Liu is utilized for its teaching of a sentence-based 
query format, wherein a sentence is processed using a parser (Col. 5, Lines 45-65). The 
applicants further argue that Hermes teaches away from such a natural language process 
(Amendment, Page 8). In response, the examiner notes that the text processing in Liu would fall 
within the goal discussed in Hermes, since it is merely a well-known extension of the text 
processing in Hermes and is commonly used in NL processing. The examiner also points out 
that Hermes further encourages the use of natural language in image retrieval (Section 4). Thus, 
Hermes does not teach away from NL sentence processing, as is argued by the applicants. 

Thus, for at least the above reasons, the independent claims remain rejected. 

The dependent claims are argued as further limiting rejected independent claims. In 
response, see the above reply directed towards the independent claims. 

Claim Rejections - 35 USC § 103 

4. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 
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5. Claims 1-3 and 5-8 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Hermes et al ("Image Retrieval for Information Systems, " 1995) in view of Liu et al (U.S. 
Patent: 6,970,860). 

With respect to Claims 1, 5, and 7, Hermes discloses: 

A dictionary storing means for storing a dictionary used for processing a natural language 
(stored natural language text descriptions associated with image features, Page 396, Section 2); 

A color/shape threshold storing means for storing color histograms mapped to color 
related words and edge information corresponding to shape related words (color histograms and 
related contour data associated with search terms stored in the IRIS system, Pages 396-399, 
Sections 2. 1-2.1. 3; and Fig. 2) ; 

A query input means for receiving a query that describes the color and the shape of the 
image by using a natural language (input query in a natural language to the IRIS system, Page 
395, Section 2, Page 404, Section 4; and Fig. 8)\ 

An analyzing means for analyzing the query sentence based on the dictionary information 
and generating analyzed words (query processing (analysis) to retrieve an image, Fig. 8; and 
Page 403, Section 3)\ 

A color/shape recognizing means for recognizing whether the analyzed words represent 
the color or the shape (color and shape based query processing, Fig. 8)\ 

A color/shape threshold database constructing means for mapping and storing color 
histograms to color related words and storing edge information corresponding to shape related 
words (IRIS database that maps image features including shape and color to user concepts in a 
natural language for image retrieval, Pages 395-400, Sections 2-2.3); 
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A color/shape threshold retrieving means for retrieving the color histograms and the edge 
information corresponding to the analyzed words from the color/shape threshold storing means 
and retrieving an image satisfying the retrieved color histograms and edge information (query 
processing that retrieves an image corresponding to color and edge information in the IRIS 
system, Pages 403-404, Section 3; and Fig. 8)\ and 

A retrieving result output means for providing image data searched in the color/shape 
threshold retrieving means (query results and corresponding thumbnail images, Fig, 8). 

Although Hermes discloses the ability of a user to input a natural language query to 
locate a desired image, Hermes does not explicitly note that such a query may be in the form of a 
natural language sentence. Hermes also fails to disclose a means for analyzing such a sentence. 
Liu, however, discloses a process for enabling a user to provide a natural language input to a 
search engine for image retrieval, wherein the input is in the form of a natural language sentence 
(natural language text entry area, Col 5, Lines 45-55). Liu also recites a natural language parser 
that identifies keywords through syntactic and semantic information in the input sentence and 
utilizes the keywords in a matching process to retrieve an image (NL parser, Col 5, Lines 56-65; 
and the matching of extracted keywords to histogram, shape, and texture information associated 
with an image, Col 6, Lines 38-58). 

Additionally, with respect to claim 7, Liu recites an image retrieval method 
implementation as a program stored on a computer readable medium (Col 4, Lines 19-31). 

Hermes and Liu are analogous art because they are from a similar field of endeavor in 
image retrieval systems. Thus, it would have been obvious to a person of ordinary skill in the art, 
at the time of invention, to modify the teachings of Hermes with the natural language sentence 
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parsing elements taught by Liu in order to provide a means for better understanding and 
identifying search terms in an image retrieval process (Liu, Col 5, Lines 56-65). 

With respect to Claims 2, 6, and 8, Liu discloses image annotation when no pertinent 
images are retrieved (Col. 9, Lines 18-23; and Col 5, Lines 19-25), while Hermes recites query 
terms related to color and shape as applied to Claim 1 . 

With respect to Claim 3, Hermes further discloses: 

The color/shape threshold constructing means maps the word representing the color to the 
color histogram and stores the word representing the color mapped to the color histogram 
(mapped image features associated with a color histogram, Pages 396-397, Section 2 A), the 
word not representing the color but reminding the color along with the corresponding color 
histogram (additional words mapped to a color histogram, Page 403, Section 3) and the edge 
information corresponding to the shape related word (mapped contour (shape) features for image 
retrieval, Pages 398-399, Section 2.1.3). 

6. Claim 4 is rejected under 35 U.S.C. 103(a) as being unpatentable over Hermes et al in 
view of Liu et al and further in view of Jain-et al (U.S. Patent: 5,983,237). 

With respect to Claim 4, Hermes in view of Liu teaches the image query system utilizing 
color and shape information, as applied to claim 3. Hermes in view of Liu do not specifically 
disclose the use of a qualification relation, however Jain teaches such a relation (Col 9, Lines 
12-24). 

Hermes, Liu, and Jain are analogous art because they are from a similar field of endeavor 
in image retrieval systems. Thus, it would have been obvious to a person of ordinary skill in the 
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art, at the time of invention, to modify the teachings of Hermes in view of Liu with the 
qualification relations taught by Jain in order to eliminate unnecessary visual senses when 
constructing a query (Jain, Col 9, Lines 18-20). 

Conclusion 

7. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the mailing 
date of this final action. 

8. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to James S. Wozniak whose telephone number is (571) 272-7632. 
The examiner can normally be reached on M-Th, 7:30-5:00, F, 7:30-4, Off Alternate Fridays. 



Application/Control Number: 10/025,912 



Page 9 



Art Unit: 2626 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Patrick Edouard can be reached at (571) 272-7603. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 



James S. Wozniak 
5/16/2007 




